1. Field of the Invention
The present invention relates to computerized simulation of what are known as giant reservoirs with automatic recovery from system failures in a parallel computing platform composed of a number of processor nodes.
2. Description of the Related Art
In the oil and gas industries, massive amounts of data are required to be processed for computerized simulation, modeling and analysis for exploration and production purposes. For example, the development of underground hydrocarbon reservoirs typically includes development and analysis of computer simulation models of the reservoir. These underground hydrocarbon reservoirs are typically complex rock formations which contain both a petroleum fluid mixture and water. The reservoir fluid content usually exists in two or more fluid phases. The petroleum mixture in reservoir fluids is produced by wells drilled into and completed in these rock formations.
A geologically realistic model of the reservoir, and the presence of its fluids, also helps in forecasting the optimal future oil and gas recovery from hydrocarbon reservoirs. Oil and gas companies have come to depend on geological models as an important tool to enhance the ability to exploit a petroleum reserve.
Reservoir simulators such as POWERS and GigaPOWERS have been described in the literature. See, for example articles by Dogru, A. H., et al.: “A Parallel Reservoir Simulator for Large-Scale Reservoir Simulation,” SPE Reservoir Evaluation & Engineering Journal, pp. 11-23, 2002, by Dogru, A. H. et al., “A Next-Generation Parallel Reservoir Simulator for Giant Reservoirs,” SPE 119272, proceedings of the 2009 SPE Reservoir Simulation Symposium, The Woodlands, Tex., USA, Feb. 2-4, 2009 and by Dogru, A. H., Fung, L. S., Middya, U., Al-Shaalan, T. M., Byer, T., Hoy, H., Hahn, W A., Al-Zamel, N., Pita, J., Hemanthkumar, K., Mezghani, M., Al-Mana, A., Tan, J, Dreiman, T., Fugl, A, Al-Baiz, A., “New Frontiers in Large Scale Reservoir Simulation,” SPE142297, Proceedings of the 2011 SPE Reservoir Simulation Symposium, The Woodlands, Tex., USA, Feb. 21-23, 2011.
In simulation models, the reservoir is organized into a number of individual cells. Seismic data with increasing accuracy has permitted the cells to be on the order of 25 meters areal (x and y axis) intervals. For what are known as giant reservoirs, the number of cells is the least hundreds of millions, and reservoirs of what is known as giga-cell size (a billion cells or more) are encountered.
An example reservoir of the type for which production data are simulated over the expected reservoir life as illustrated by the model M (FIG. 1) is usually one which is known to those in the art as a giant reservoir. A giant reservoir may be several miles in length, breadth and depth in its extent beneath the earth and might, for example, have a volume or size on the order of three hundred billion cubic feet.
Simulation of giant reservoir models is possible only on large computing platforms where simulation task is parallelized. In parallel computation, the simulation model is divided into many small partitions, and every partition is assigned to a specific computing element or processor node.
Giant reservoir simulation models have been built which required large computational resources. It typically has taken several days to complete a simulation. These simulations are done by high performance computing (HPC) computer clusters, which are groups of processor nodes. The processor nodes are available from several sources. During simulation of a giant reservoir, the reservoir model is decomposed or partitioned into a number of subdomains, and each of the processor nodes is assigned processing of a particular subdomain of cells of the reservoir model.
So far as is known, during reservoir simulation according to the prior art, a simulation engineer submitted the simulation job to the data processing system. The simulation engineer was then required to monitor the simulation job while it was being performed by monitoring progress of the job through the data processing system user interface.
It was possible that such reservoir simulation jobs could fail because of processing system problems. Failure of a single processor in the pool of processors responsible for computing a simulation job was likely to cause failure of the simulation job. In the event of a processing system problem in even a single processor node during a reservoir simulation, the entire simulation job would effectively be lost, and the time spent in the failed simulation was lost. The user was required to manually resubmit the simulation job which had been lost, and restart the simulation job from the beginning in most cases. Many simulation engineers lost significant amounts of productive time because of job failures on simulation clusters.
Processor failure rate have been observed on current high performance computing platforms to be about 0.2 failures per year per node. Such a processor failure rate translates into failure of many jobs which may be 1% or more simulation jobs per year. As more massive reservoir models are built, more nodes are required to be used for a simulation. In addition, simulation runs are projected to take longer time to finish for such reservoir models.